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Program description 



The Connected Mathematics Project (CMP) is a problem- 
centered mathematics curriculum designed for all students in 
grades 6-8. Each grade level of the curriculum Is a full-year 
program and covers numbers, algebra, geometry/measurement, 
probability, and statistics. The program seeks to make con- 
nections within mathematics, between mathematics and other 
subject areas, and to the real world. The curriculum is divided 
into a sequenced set of units, each organized around different 



mathematical topics. The four to seven lessons in a unit each 
contain one to five problems that the teacher and students 
explore in class. Additional problem sets, called Applications, 
Connections, and Extensions, in each lesson help students 
practice, apply, connect, and extend their understanding and 
skills. Each lesson culminates in a Mathematical Reflections 
activity. According to the developers, the CMP addresses 
National Council of Teachers of Mathematics standards. 



RSSSarch Three studies of the CMP met the What Works Clearinghouse schools located in northeastern, south central, midwestern, and 

(WWC) evidence standards with reservations.^ The three studies western states, 

included over 26,000 students from grades 6-8 in about 100 



Effectiveness The CMP curriculum was found to have mixed effects on math achievement. 



Math achievement 

Rating of effectiveness Mixed effects 
Improvement index^ Average: -2 percentile points 

Range: -12 to +11 percentile points 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and the range of improvement indices for all findings across two of the three studies. One additional study that showed 
a statistically significant positive effect is not included in the average and range. 
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Updating Previous Research 

This report updates the previous WWC report on CMP that 
was released on the WWC website in November 2004. Since 
the release of the previous report, the WWC has updated its 
evidence standards and deveioped peer-reviewed procedures 
for addressing certain methodological flaws in original studies, 
such as mismatch between the unit of assignment and the unit 
analysis and lack of adjustment for multiple comparisons. These 
standards and procedures, when applicable, have been applied 
to studies included in the original CMP review. No new studies 
were identified for this updated report. 

Developer and contact 

The CMP was developed at Michigan State University by Glenda 
Lappan, James T. Fey, William F. Fitzgerald, Susan N. Friel, 
and Elizabeth D. Phillips. Email: cmp@math.msu.edu. Web: 
http://connectedmath.msu.edu . Telephone: (517) 432-2870. The 
curriculum is distributed by Pearson Prentice Flail. Web: http:// 
phcatalog.pearson.com . 

Scope of use 

Pilot editions of CMP were used between 1991 and 1997 by 
approximately 160 teachers and 45,000 students in diverse set- 
tings across the United States. As of September 2004, it had been 
implemented in 2,462 school districts, covering all 50 states. 

Teaching 

This problem-centered curriculum is based on an inquiry model 
of instruction, which consists of three phases: launch, explore. 



and summarize. In the first phase, the teacher launches the 
problem with the whole class, introduces new ideas, clarifies 
definitions, reviews old concepts, and connects the problem 
to students’ past experiences. In the explore phase, students 
work individually, in pairs or small groups, or occasionally as 
a whole class to solve the problem. In the summarize phase, 
students discuss their solutions as well as the strategies that 
they used to approach the problem, organize the data, and find 
the solution. 

Intended as a three-year mathematics curriculum, CMP 
covers grades 6-8, providing eight student units for each 
grade level. Each student unit is organized around an important 
mathematical idea or cluster of related ideas and is divided 
into several investigations, with each investigation containing 
a series of problems. The implementation plan is based on a 
45-60 minute class period and a 180-day school year. The CMP 
provides teacher guides specifically designed for each student 
unit. The teacher guides include discussions of the mathematics 
of the unit, instructional strategies, and assessment resources. 
The developer suggests that when a district uses the curriculum 
for the first time, it should establish a support system to all the 
CMP teachers in a building. 

Cost 

According to Pearson Prentice Hall, the publisher, the most 
recent edition of the CMP costs $8.47 per student and 
$20.97 per teacher unit. See the publisher for costs for other 
resources. 



Twenty-two studies reviewed by the WWC investigated the 
effects of CMP. Three studies (Ridgway, Zawojewski, Hoover, & 
Lambdin, 2002; Riordan & Noyce, 2001;^ Schneider, 2000) were 
quasi-experimental designs that met WWC evidence standards 



with reservations. The remaining 19 studies did not meet WWC 
evidence screens. 

Ridgway, Zawojewski, Hoover, & Lambdin (2002) included stu- 
dents in grades 6-8 from 18 schools located in the midwestern. 



3. Riorden & Noyce (2001) also examined effects of the program Everyday Mathematics'^. For further details of this analysis see the Everyday Mathematics ^ 
Intervention report. 
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The WWC found the 
Connected Math&natics 
Project to have mixed effects 
for math achievement 
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western, and eastern regions of the country. Students using the 
CMP curriculum were compared to students who did not use the 
curriculum."^ 

Riordan & Noyce (2001) included eighth-grade students 
from 50 schools in Massachusetts. Students using the CMP 
curriculum were compared to students who did not use the 



CMP program, but used different published textbook programs, 
which, in the aggregate, represented the instructional norm in 
Massachusetts. 

Schneider (2000) included three cohorts of middle school stu- 
dents from 48 schools in Texas. Students using the CMP curricu- 
lum were compared to students who did not use the curriculum. 



Findings 

The WWC review of interventions for middle school math 
addresses student outcomes in one domain: math achievement. 

The Ridgway, Zawojewski, Hoover, & Lambdin (2002) study 
examined students’ scores on the Iowa Test of Basic Skills (ITBS) 
and reported a statistically significant negative effect, favoring the 
comparison group; however, the WWC analysis did not confirm the 
statistical significance of this outcome. The study also examined 
total scores on the Balanced Assessment Test and reported statis- 
tically significant positive effects; however, this was not confirmed 
by the WWC. The average effect size for math achievement across 
study findings was not large enough to be considered substan- 
tively important. So, in this study, CMP had an indeterminate effect 
on math achievement, according to WWC criteria. 

The Riordan & Noyce (2001) study examined total scores on 
the Massachusetts Comprehensive Assessment System (MCAS) 
and reported a statistically significant positive effect. 

The Schneider (2000) study examined passing rates and 
students’ scores on the Texas Learning Index using the Texas 



Assessment of Academic Skills (TAAS) and found no statistically 
significant effects. In addition, the average effect size across all 
outcomes for this study was neither statistically significant nor 
substantively important (that is, at least 0.25). So, in this study, 
CMP had an indeterminate effect on math achievement, accord- 
ing to WWC criteria. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings (as calculated 
by the WWC®), the size of the difference between participants 
in the intervention and the comparison conditions, and the con- 
sistency in findings across studies (see the WWC Intervention 
Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 



an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations) . The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 



4. The WWC reviewed findings for students in grade 6 oniy as baseiine data (that is, math achievement before exposure to the program took place) was not 
taken into account in this study. 

5. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of CMP, corrections for clustering and multiple com- 
parisons were needed. 
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The WWC found the 
Connected Mathematics 
Project to have mixed 
effects for math 
achievement (continued) 



the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is entirely based on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analysis. The improvement index can take on values between 
-50 and +50, with positive numbers denoting favorable results. 
The average improvement index for math achievement is -2 
percentile points across the three studies, with a range of -12 to 
+11 percentile points across findings in two of the three studies. 
One additional study that showed a statistically significant posi- 



tive effect is not included in the average and range because a 
student-level improvement index could not be computed. 

Summary 

The WWC reviewed 22 studies on the Connected Mathematics 
Project. Three of these studies met WWC evidence standards 
with reservations; the remaining studies did not meet WWC evi- 
dence screens. Based on these three studies, the WWC found 
the program to have mixed effects on math achievement. The 
evidence presented in this report may change as new research 
emerges. 
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Appendix 



Appendix A1.1 


Study characteristics: Ridgway, Zawojewski, Hoover, & Lambdin, 2002 (quasi-experimentai design) 


Characteristic 


Description 


Study citation 


Ridgway, J. E., Zawojewski, J. S., Hoover, M. N., & Lambdin, D. V. (2002). Student attainment in the Conneoted Mathematics curricuium. In S. L. Senk & D. R. Thompson 
(Eds.), Standards-based school mathematics curricula: What are they? What do students learn? MaUmh, N.J.: Lawrence Erlbaum Associates, Inc. 


Participants 


The 1994/95 sample included over 340 sixth graders and 630 seventh graders from 9 CMP schools (2 classrooms per grade from each school), and 160 sixth graders and 
250 seventh graders from 9 comparison schools (1 classroom per grade from each school).'' The 1995/96 sample included over 780 eighth-graders from an unspecified 
number of CMP schools and 300 eighfh graders from an unspecified number of comparison schools. Some students were included in both the 1994/95 sample and the 
1995/96 sample. Demographic characteristics of fhe participants are not reported. The WWC review only included findings from the sixth grade, because baseline data were 
available only for the sixth-grade students and not for the other grade levels. 


Setting 


Participating classrooms were from schools located in the midwestern, western, and eastern regions of the country. 


Intervention 


Teachers in the intervention group were using the CMP as the core curriculum throughout the school year. The study authors did not report how the program was actually 
implemented in those classrooms. All the sixth-grade students in the study were new to the program, and about three-fourths of the seventh- and eighth-grade students in the 
study had used the program in the previous year. 


Comparison 


Teachers in the comparison group did not implement the CMP, nor were they involved in any reform efforts. Data were not available about the mathematics textbook series 
used by those teachers. 


Primary outcomes 
and measurement 


Math achievement was assessed using the Iowa Test of Basic Skills (ITBS) Survey Battery and Balanced Assessment (BA) Test (see Appendix A2 for more detailed descrip- 
tions of the outcome measure). 


Teacher training 


All CMPteachers attended the summer CMP institutes at Michigan State University. 



1 . The WWC requested and received from one of the study authors information about the number of schoois that participated in the study. 
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Appendix A1.2 Study characteristics: Riordan & Noyce, 2001 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Riordan, J. E., & Noyce, P. E. (2001). The impact of two standards-based mathematics curricuia on student achievement in Massachusetts. Journal for Research in Mathemat- 
ics Education, 32(4), 368-398. 


Participants 


This study iooked at 20 CMP schools with 1,879 eighth-graders and 30 matched comparison schoois with 4,978 eighth-graders. Overaii, 10% of the student participants 
were eligible for free or reduced-price iunches, and 87% were whife. Aii sfudenfs were reguiar educafion studenfs.^ 


Setting 


This sfudy included relatively advantaged middle schools with predominantly white students and a low percentage of students receiving free or reduced-price lunches in 
Massachusetts. 


Intervention 


Schools in the intervention group had implemented at least 11 student units in grades 6-8 by 1998/99, but none of the schools implemented all eight units that the CMP has 
available for each grade. Eurther, it is not clear how the program was actually implemented in those schools. The 20 schools in the intervention group had implemented the 
program for two to three years. 


Comparison 


The 30 comparison schools did not implement the CMP, but used 15 different textbook programs, which, in the aggregate, represented the instructional norm in Massachu- 
setts. The most commonly used programs were those published by Eleath, Addison-Wesley, Prentice Hall, and Houghton-Mifflin. 


Primary outcomes 
and measurement 


Math achievement was measured using the Massachusetts Comprehensive Assessment System (see Appendix A2 for more detailed descriptions of the outcome measure). 


Teacher training 


No information about teacher training was provided. 



1. This study also included an additional separate examination of a single intervention school that has been using the program for four years and its four comparison schools. Because the results of 
students in the single intervention schools were confounded with the school site, this analysis was not included in the WWC review. 
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Appendix A1.3 Study characteristics: Schneider, 2000 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Schneider, C. L. (2000). Connected Mathematics and the Texas Assessment of Academic Skills. Dissertation Abstracts internationai, 6?(12), 4709A. (UMi No. 9997642). 


Participants 


The study included 3 cohorts from 23 C/WP schools and 25 matched comparison schools overall. Cohort 1 was from 23 intervention and 19 comparison schools. Cohort 
2 was from 22 intervention and 19 comparison schools. Cohort 3 was from 18 intervention and 18 comparison schools. For TAAS TLI scores, data was collected from 16 
schools per condition (CMP, comparison) and per cohort (that is, cohort 1, cohort 2, cohort 3), because six schools were omitted from the analysis due to missing data or 
because the intervention was discontinued. Not all cohorts participated in every analysis because different grade levels were followed through for each cohort. 


Setting 


The participating schools were located in rural, suburban, and urban and both low and high socioeconomic areas of Texas. Those schools varied in the racial composition, 
socioeconomic status, special education status, and English language learner status of the student populations that they served. Many of the schools had predominantly 
minority student populations. 


Intervention 


Schools in the intervention group were using the C/WPfor grades 6-8. There were substantial variations in the extent to which the curriculum was used at each grade and 
each year across these schools. The three cohorts in the intervention group received the C/WP for three years, two years, and one year, respectively, between 1996/97 and 
1998/99. 


Comparison 


The 25 comparison schools did not implement the CMP, and it is unclear what mathematics curricula they were using. 


Primary outcomes 
and measurement 


Students' mathematic achievement was measured using the Texas Assessment of Academic Skills (TAAS) passing rate and Texas Learning Index (TLI) (see Appendix A2 for 
more detailed descriptions of the outcome measures). 


Teacher training 


Teachers who taught grade 6, 7, or 8 at the 23 C/WP schools participated in a six-day summer professional development conducted by the Texas Statewide Systemic Initiative 
in 1996, 1997, and 1998. 
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Appendix A2 Outcome measures in the math achievement domain 



Outcome measure 


Description 


Iowa Test of Basic 
Skills (ITBS) 


The ITBS is a norm-referenced state standardized test. The mathematics sections measure the following math abilities: problem solving, data interpretation, math concepts, 
estimation, and computation (as cited in Ridgway, Zawojewski, Hoover, & Lambdin, 2002). The Ridgway et al. (2002) study used the ITBS Survey Battery Form K (levels 12,13, 
and 14). 


Balanced Assessment 
(BA) Test 


The BA test was designed to assess students’ math achievement in a variety of curricular areas through constructed-response items that require a range of responses from 
short answer to extended response. It was developed through the collaboration between the CMP developer and the Balanced Assessment Project (as cited in Ridgway et al., 
2002). 


Massachusetts 
Comprehensive Assessment 
System (MCAS) 


The MCAS is a criterion-referenced state standardized test that includes both multiple-choice and open-response questions (as cited in Riordan & Noyce, 2001). 


Texas Assessment 
of Academic Skills 
(TAAS) — passing rates 


The TAAS is a criterion-referenced state standardized test that measures problem-solving and critical-thinking skills. The passing rate is the percentage of students that 
reached a proficient level (as cited in Schneider, 2000). 


Texas Assessment of 
Academic Skills (TAAS) — 
Texas Learning Index (TLI) 


The TAAS is a criterion-referenced state standardized test that measures problem-solving and critical-thinking skills. TLI is a TAAS-based statistic designed for comparing 
student progress between administrations and between grades (as cited in Schneider, 2000). 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


CMP Comparison 

groups group 


Mean difference^ 
[CMP- 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index^ 



Ridgway, Zawojewski, Hoover, & Lambdin, 2002 (quasi-experimental design)^ 



ITBS 


Grade 6 


18/500 


8.10 

(2.30) 


8.60 

(2.70) 


-0.50 


-0.20 


ns 


-8 


Balanced Assessment Test 


Grade 6 


18/500 


21.50 

(12.60) 


18.10 

(11.90) 


3.40 


0.27 


ns 


+11 


Average® for math achievement (Ridgway, Zawojewski, Hoover, & Lambdin, 2002) 






0.04 


ns 


+1 








Riordan & Noyce, 2001 (quasi-experimental design)® 








MCAS 


Grade 8 


50/6,857 


238.20 

(9.08) 


233.85 

(10.57) 


4.35 


na^® 


Statistically 

significant 


na^® 


Average® for math achievement (Riordan & Noyce, 2001) 








na^® 


Statistically 

significant 


na^® 








Schneider, 2000 (quasi-experimental design)® 








TAAS TLI score 


Grade 6-8 
(Cohort 1) 


32/5,701 


73.20 

(6.37) 


72.90 

(5.46) 


0.30 


0.05 


ns 


+2 


TAAS TLI score 


Grades 6-7 
(Cohort 2) 


32/6,461 


72.30 

(6.59) 


74.20 

(5.29) 


-1.90 


-0.32 


ns 


-12 


TAAS TLI score 


Grade 6 
(Cohort 3) 


32/7,339 


73.60 

(5.41) 


74.50 

(6.37) 


-0.90 


-0.15 


ns 


-6 


Average® for math achievement (Schneider, 2000) 










-0.14 


ns 


-6 


Domain average® for academic achievement across all studies 








-0.05 


na 


-2 



ns = not statistically significant 
na = not applicable 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices. Additional findings are presented in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Means and 
standard deviations for the Riordan & Noyce (2001) and Schneider (2000) studies were received from the study authors. 

3. In the Ridgeway et al. (2002) study, the CMP group mean equals the comparison group mean plus the mean difference. The computation of the mean difference took into account the pretest difference between the study groups. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ (continued) 



7. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition and the percentiie rank of the average student in the comparison condition. The improvement index can 
take on vaiues between -50 and +50, with positive numbers denoting favorabie resuits. 

8. The ievei of statisticai significance was reported by the study authors, or where necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie comparisons. For an expianation about the ciuster- 
ing correction, see the WWC Tutoriai on Mismatch . See Technicai Detaiis of WWC-Conducted Computations for the formuias the WWC used to caicuiate statisticai significance, in the case of Ridgway, Zawojewski, Hoover, & Lambdin 
(2002) corrections for ciustering and muitipie comparisons were needed, so the significance ieveis may differ from those reported in the originai study, in the case of Schneider (2000), corrections for ciustering and muitipie compari- 
sons were not needed. Riordan & Noyce (2001) examined student outcomes for iow socioeconomic status (SES) students and for maie and femaie students separateiy, and reported statisticaiiy significant positive effects favoring the 
C/WP group for iow-SES and femaie students. The resuits of these anaiyses were not inciuded for rating purposes because of overiap with resuits for the entire sampie. in addition, standard deviations were not avaiiabie for this subset of 
findings. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simpie averages rounded to two decimai piaces. The average improvement indices are caicuiated from the average effect size. 

10. Student-ievei standard deviations were not avaiiabie for this study. Schooi-ievei standard deviations were 9.08 for the intervention group and 10.57 for the comparison group. Because the student-ievei effect size and improvement in- 
dex couid not be computed, the magnitude of the effect size was not considered for rating purposes. However, the statisticai significance for this study is comparabie to other studies and is inciuded in the intervention rating. For further 
detaiis, piease see Technicai Detaiis of WWC-Conducted Computations . 
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Appendix A4 Summary of additional findings for the math achievement domain^ 



Authors’ findings from the study 



Mean outcome 

(standard deviation^) WWC calculations 



Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


CMP 

group 


Comparison 

group 


Mean difference^ 
{CMP- 
comparison) 


Effect size^ 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 




Schneider, 2000 (quasi-experimental design^ 


TAAS pass rates 


Grade 8 (Cohort 1) 


48/1,440 


80 percent of 


86 percent of 


0.65 


-0.26 


ns 


-10 








students 


students 
















(na) 


(na) 










TAAS pass rates 


Grade 7 (Cohort 2) 


42/1,260 


76 percent of 


83 percent of 


0.65 


-0.26 


ns 


-10 








students 


students 
















(na) 


(na) 










TAAS pass rates 


Grade 6 (Cohort 3) 


42/1,260 


78 percent of 


82 percent of 


0.78 


-0.15 


ns 


-6 



students students 

(na) (na) 



ns = not statistically significant 

na = not applicable 

1 . This appendix presents additional findings for the TAAS measure. Findings on TLI scores generated from the same measure were used for rating purposes and are presented in Appendix A3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. Effect sizes for the binary outcome measure of TAAS pass rates in Schneider (2000) were calculated using the odds ratio formula and converted to standardized mean difference. For an explanation of the effect size calculation, see 
Technical Details of WWC-Con c!uctod Co mputations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can 
take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings 
not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate 
statistical significance. In the case of Schneider (2000), no correction for clustering was needed. 
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Appendix A5 Connected Mathematics Project rating for the math achievement domain 



The WWC rates the effects of an intervention for a given outcome domain as: positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.^ 
For the outcome domain of math achievement, the WWC rated CMP as having mixed effects. It did not meet the criteria for positive effects or potentially positive 
effects because only one study showed a statistically significant positive effect and two studies showed indeterminate effects. The remaining ratings (no discernible 
effects, potentially negative effects, or negative effects) were not considered as CMP was assigned the highest applicable rating. 

Rating received 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No studies showed a statistically significant or substantively important negative effect. 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Met. One of the three studies meeting standards with reservations showed a statistically significant positive effect; the remaining two studies 
showed an indeterminate effect. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies met WWC evidence standards for a strong design. 

• Criterion 2; No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study that met standards with reservations showed a statistically significant positive effect. 

• Criterion 2; No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. No studies showed statistically significant or substantively important negative effects. Two of the three studies showed indeterminate 
effects, and only one study showed a statistically significant positive effect. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings 
of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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